Calibrating Vulnerability Scores with Expert Opinions

class: center, middle, inverse, title-slide

.title[
# Calibrating Vulnerability Scores with Expert Opinions
]
.subtitle[
## When, Why and How!
]
.author[
### DIMA, Americas Bureau
]
.date[
### 10 November 2022
]

---

# What are vulnerability scores?

Similarly to credit score in the private sector, vulnerability score offers a way to combine in a single number (i.e. a score for instance between 1 & 100) a composite measurement of vulnerability status in order to assess if a person can qualify for assistance

---

# When do you need to use vulnerability scores?

*  Necessary once the population size exceed a certain scale – case by case management is not do-able or comes with high risk of exclusion

*  Minimize subjective component in assistance allocation

*  Measurement allow to prioritize assistance (Cash, but also potentially training, livelihood, micro-credit, etc.) in function of available budget

---

# When not all eligible people can be prioritised....

.pull-left[
 In protracted situations, not all eligible persons can be prioritised...
]

.pull-right[

![https://openknowledge.worldbank.org/bitstream/handle/10986/37228/9781464818141.pdf](data:image/png;base64,#eligible.png)

]

---

# When do you need to use expert opinions?

.pull-left[
 When we can not use simpler targeting approach - that does not require to distinguish eligibility from prioritization
 
 When we have experts....
]

.pull-right[

![ ](data:image/png;base64,#impartial.jpg)
]

---

# When there's no representative dataset available

When we have dataset that represents statistically well the whole population, it is possible to use statistical model, either:

* proxy means testing: aka one outcome indicator can be used as single proxy of the vulnerability situation of the case (poverty, deprivation, etc.. )-> supervised classification
 
 * cluster population in consistent profiles that comes out from the data -> unsupervised classification - Item response Theory

But in most humanitarian situation, achieving good statistical representativenesss is challenging (volatile environment, hidden population to sample..)

---

# What are the challenges with expert opinions?

* agree on the relevant eligibility criteria
 
 * agree on the relative importance of each criteria
 
 * agree on what other are agreeing...
 
 
Without a specific facilitation approach, this can end into lengthy discussions...

---

# Main challenges to create vulnerability scores

Are selected indicators “add-able” in some sensical way, given the real-world meaning of the indicators?

Can you add apples and oranges?

![](data:image/png;base64,#apple.png)

---

# The monkey dilemna: How to combine criteria? aka "compensability" for dummies

![](data:image/png;base64,#compensability1.png)

---

# Compensavility does not allow to reflect interactions between criteria

![](data:image/png;base64,#compensability2.png)

---

# An organised workshop facilitation

6 simple steps:

* Step 1- Expert training

* Step 2- Select the relevant eligibility criteria: quadratic voting

* Step 3- Adjust the weight of criteria: conjoint analysis

* Step 4- Review results and potentially iterate

* Step 5- Implement the formula in the vulnerability scoring form

* Step 6 - Apply Eligibility and prioritization threshold

---

# Step 1- Expert training

>  "It is not enough to do your best; you must know what to do, and then do your best."
>
> W. Edwards Deming

- Understand the difference between output variable and eligibility criteria

---

# Step 2- Select the relevant eligibility criteria: quadratic voting

![https://docs.wfp.org/api/documents/WFP-0000122035/download/](data:image/png;base64,#outcome.jpeg)

[Technical step by step tutorial: set up a quadratic vote on Kobotoolbox](quadraticVote.html)

---

# Step 3- Adjust the weight of criteria: conjoint analysis

Conjoint analysis is a type of consultation designed to measure the average opinion from a group of experts through specific pool where experts should compare different stereotypical profiles one by one and assess their respective vulnerability level.

**Quotes:**

>  "If we have data, let’s look at data. If all we have are opinions, let’s go with mine!"
>
> Jim Barksdale

[Technical step by step tutorial: set up a conjoint analysis on Kobotoolbox](conjointAnalysis.html)

---

# Step 4- Review results and potentially iterate

---

# Step 5- Implement the formula in the vulnerability scoring form

Kobotoolbox form are often the default tool: The average weight obtained through conjoint analysis are implemented through a calculated filed using pow function

Do not display the score during the screening - only provide information on eligibility

---

# Step 6 - Eligible and prioritised

Final scores are compared with 2 distinct threshold:

- Eligibility threshold - applicants that should be covered based on their needs profile

- Prioritization threshold - applicants that should be covered based on the available budget for the current assistance cohort

---

# Conclusion

When there's no representative data, expert opinion is the default options

Creating buy-in on how to calibrate the vulnerability scoring formula is a key to the social acceptability

Combining organised consultation (quadratic voting and conjoint analysis) allows to leverage collective intelligence

As soon as you use a scoring system, the management of fluctuating prioritization threshold (i.e budget...) implies an efficient case management system:
 - Assistance cohort management
 - Continuous Vulnerability scoring
 - Appeal system